Feature extraction, storage, running time (cost)

Submission Time

Dimensions of Extracted Features

Baseline Model

Advanced Model

Submitted File Sizes

Performance

Prediction Accuracy

Computational Efficiency

Team 9 is the only team that implemented model selection within the training process, which takes much longer time. However, to be fair in the comparison, their average training CPU time per fold per replicate is divided 7, which is the number of their candidate parameter values. They’ve also performed a 5-fold CV within each GBM training.

Fun Images

The percentages of votes to cat (1) of 18 fun images.

## NULL

Team Performance Clustering

We use Multidimensional Scaling (MDS) to cluster the performance of each team using the prediction in the first replicate.

Retrained Advanced Model

## $points
##              [,1]          [,2]
##  [1,] -0.05827984  0.3379870516
##  [2,]  0.08787685 -0.0766731754
##  [3,]  0.16314396 -0.0325527129
##  [4,] -0.17510701  0.2040679592
##  [5,]  0.10442062 -0.0406562996
##  [6,]  0.09791828 -0.0008004062
##  [7,] -0.29916904 -0.1074944152
##  [8,] -0.27641054 -0.1846644649
##  [9,]  0.10278298 -0.0552844831
## [10,]  0.09215284 -0.0195873549
## [11,]  0.16067089 -0.0243416987
## 
## $eig
##  [1] 2.996655e-01 2.141596e-01 1.470708e-01 1.189221e-01 6.357198e-02
##  [6] 4.980865e-02 2.556054e-02 1.519005e-02 1.142172e-02 9.986528e-03
## [11] 1.386641e-17
## 
## $x
## NULL
## 
## $ac
## [1] 0
## 
## $GOF
## [1] 0.5378354 0.5378354

Untuned Advanced Model

## $points
##             [,1]        [,2]
## [1,] -0.13557125  0.36873194
## [2,]  0.12319907 -0.04894373
## [3,] -0.21887867 -0.12494272
## [4,]  0.13957533 -0.05057108
## [5,]  0.02061158 -0.04208791
## [6,]  0.20248544  0.04288016
## [7,]  0.11984681 -0.02299910
## [8,] -0.25126831 -0.12206756
## 
## $eig
## [1] 2.198709e-01 1.755664e-01 4.509765e-02 3.899978e-02 2.717722e-02
## [6] 1.775751e-02 2.020690e-03 3.418264e-17
## 
## $x
## NULL
## 
## $ac
## [1] 0
## 
## $GOF
## [1] 0.7510821 0.7510821

Summary of Common Problems